Tensor Products of Singular Representations and 
an Extension of the ^-correspondence 

Alexander Dvorsky and Siddhartha Sahi* 

Department of Mathematics, Rutgers University, New Brunswick 

NJ 08903, USA 



Abstract 

In this paper we consider the problem of decomposing tensor products 
of certain singular unitary representations of a semisimple Lie group G. 
Using explicit models for these representations (constructed earlier by 
one of us) we show that the decomposition is controlled by a reductive 
homogeneous space G'/H'. Our procedure establishes a correspondence 
between certain unitary representations of G and those of G' . This extends 
the usual ^-correspondence for dual reductive pairs. As a special case we 
obtain a correspondence between certain representations of real forms of 
E-r and F4. 



Introduction 

Let F be a field and e some fixed additive character of F. If W is a finite 
dimensional vector space over F endowed with a non-degenerate skew-symmetric 
form, we can consider an associated Heisenberg group H(W). Denote by p £ 
an irreducible unitary representation of H(W) on which the center of H(W) 
operates via the character e (it is unique by the theorem of Stone and von 
Neumann). Since the symplectic group Sp(W) operates on H(W) via its action 
on the vector space W , it also acts on the representation p £ . The action is trivial 
on the center of H(W) and therefore, for any g e Sp(W), there is an operator 
u) e (g) (unique up to scaling) which intertwines p £ with pf . These operators form 
the oscillator representation of Sp(W). In general, the oscillator representation 
uj £ is projective, but it always corresponds to an ordinary representation of a 
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two-fold cover of Sp(W). This cover is denoted by Sp(W) and is called the 
metaplectic group. 

The oscillator representation originated in the works of Segal and Shale and 
was immortalized by Weil, who used it in his construction of the theta-functions 
on the metaplectic group flWfl. In one of the earliest works on the spectrum of 



the Weil (oscillator) representation, Gelbart |G1] studied the decomposition 
of the tensor product co' = cu® k , where uj is the oscillator representation of 
the real symplectic group Sp(2m,R) and k > 2m. He demonstrated that for 
k = 2m all representations of the holomorphic discrete series for Sp(2m,M.) 



occur in the spectrum of to'. Kashiwara and Vergne [KV| extended the results 



of [Gl] to tensor products , k > 1. In particular, any unitary highest weight 
representation of Sp(2m, R) will appear in the decomposition of LO® k for some 
appropriate value of k. 



Later, the approach of |G1] and [KV] was replaced by the modern approach 



to the ^-correspondence. One starts with a reductive dual pair of algebraic 
groups G and G", defined over some local field F, which are mutual centralizers 
inside a symplectic^group Sp. Let u> be an oscillator_jepresentation of the 
metaplectic group Sp on a Hilbert space H, and let E C Sp denote the preimage 
of a reductive subgroup E C Sp. Denote by 1Z(E) the set of (equivalence classes) 
of continuous irreducible representations of E on a locally convex space, which 
can be realized as quotients of H°° by cj(i?)-mvariant subspaces. 

Conjecture 0.1 (Howe's duality conjecture) The setTZ(G-G') is the graph 
of a bijection between all of TZ(G) and all oflZ(G'). Moreover, an element of 
1Z(G ■ G') occurs as a quotient of uj in a unique way. 



Howe's conjecture has been proved for F — R or C | Ho2 |, and also for 
all non-Archimedean local fields of odd residue characteristic. The resulting 
correspondence between the irreducible representations of G and those of G' is 
called the Howe duality correspondence, or the ^-correspondence. In general, 
this correspondence does not preserve unitarity, i.e., a unitary representation of 
G can correspond to a non-unitarizable representation of G' . 

If one member of a dual pair (say G') is much "smaller" than the second 
group, unitarity is preserved and the duality correspondence is a particularly 



nice one (this is a so called stable range duality | Hol |, [LlQ. We denote by 
G(e) and G'(e) the subsets of the unitary duals of G and G consisting of 
those unitary irreducible representations whose restriction to the kernel of the 
projection Sp — > Sp (i.e., the group Z2) is a multiple of the non-trivial character 
of Z2. Then the Howe correspondence gives an injection G'(e) G(e). In many 
cases the coverings G — > G and G — » G" are trivial, and we obtain an injection 
of the unitary dual of G' into that of G. 

Example. G' — U(l) and G = U(p,q) form a stable range dual pair inside 
Sp(2p+2q, R). The representations of U{p, q) appearing in the ^-correspondence 
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for this reductive dual pair are the "ladder" representations, which were given 
this name because their K-types lie along a line, i.e., their highest weights are 
obtained from the highest weight of the lowest -ftf-type by adding multiples of a 
single vector. 



The duality conjecture also has a global version [G2, 2.5], when F is a global 
field, A is an adele ring of F, and one considers a dual pair inside a symplcc- 
tic group over A. This version is of considerable representation- and number- 
theoretic interest, since the global duality correspond ence provides a way to lift 
automorphic forms between members of a dual pair |G2|] , jp|. For example, if 

G' = SL{2) and G ~ PGL(2) is realized as an orthogonal group preserving 
a 3-variables quadratic form Q(xi,X2,xz) = x\ — X2X3, the ^-correspondence 
produces the Shimura lifting which associates a modular form of weight n — 1 
to a modular form of half- integral weight ^ . 

It is a remarkable fact that every classical group and no exceptional group 
can be realized as a member of a reductive dual pair. It seems desirable to 
determine whether one can extend this theory to other reductive groups in 



a reasonable manner. In this paper we attempt to extend the original (|G1| 



[KV|) approach to the ^-correspondence. We study the decompositions of tensor 
products of certain small unitary representations of a real reductive group G 
and construct a parametrization of the spectra of these tensor products. 

Let n be a symmetric tube domain of rank n, and G — Aut(fi). The 
Shilov boundary of f2 is of the form G/P where P — LN is the Siegel-Poincare 
parabolic subgroup of G. The nilradical N is abelian and so is isomorphic to its 
Lie algebra n, and the Levi subgroup L has finitely many (coadjoint) orbits on 
n* . These orbits are indexed by their "signatures" , where a signature p consists 
of a pair of non-negative integers^ = (p + ,p~) with \p\ == p + + p~ < n. 

Every non-open orbit (with \p\ < n) has a canonical L-equivariant measure, 
and the main result of [^] is the construction of an irreducible representation of 
the universal cover of G on the associated L 2 -space. Usually these representa- 
tions descend to G, and in all cases they can be viewed as linear representations 
of a certain double cover of G, denoted by G. In this paper we consider the 
problem of decomposing certain tensor products of these representations. More 
precisely, let II = 7ri<g>- • -(g>7r s be the tensor product of representations associated 
to orbits O pir . -,0 Ps whose signatures pi = (pf ,p~[) satisfy 

(pt +Pi) + --- + (pt+Ps)<n. (1) 

Under assumption (1), which is an analogue of the stable range condition for 
the ^-correspondence, each of the following spaces contains an open and dense 
L-orbit 

O = P1 + . . . + Ps , O' = P1 x . . . x Ps . 

Fix a generic point £' = (£1, . . . , in O' such that £ = £1 + . . . is generic in 
O. We denote the inverse images of L and P in G by L and P, respectively. Let 
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S and S' be the stabilizers of £' and £ in L, and let \Z be the unitary character 
of N defined by x^(exp x) = f exp(i (£, x)) for x G n. 

Theorem 0.2 77ie restriction of II to P is isomorphic to v (g> Ind^^l (8 Xc); 
where v is a certain unitary character of P. 

In general, S 1 , S" are not reductive, however they contain reductive groups 
G" and H' and a common normal subgroup N such that S* = G' k Z and 
S' = H' K Z. We consider the direct integral decomposition 

L 2 (G'/H')= / _ TO(7r)7rd^(7r). 
For each 7r occurring in L 2 (G' / H'), we define 8(-7r) £ F by 

Q(tt) = V® IndG/ ZA r(7T (g) 1 ® xt)- 

By Mackey theory, 0(7r) is irreducible, and Theorem |0.2| implies 
Theorem 0.3 The restriction of II to P has the decomposition 

r® 

1% = / _ m(7r)e(7r)rf/i(7r) . (2) 

The main result of the paper is the following 

Theorem 0.4 For almost every tt (with respect to the Plancherel measure dfi), 
O(tt) extends to an irreducible representation of G, so that (Q) is also a G- 
decomposition. If ^ \pi\ < n, this extension is unique. 

Thus the map tt — > Q(n) gives a (measurable) bijection between unitary rep- 
resentations of G occurring in II and the unitary representations of G' occurring 
in L 2 {G'/H'). 

We now discuss some special cases of the above result: 

If s = 2, then G' / H' is a symmetric space, which is Riemannian if and only 
if Pl and P2 both have definite signatures (of the form (p + ,0) or (0,p~)). 
Positive (resp. negative) definite orbits correspond to the highest (resp. lowest) 
weight singular representations of G, and in this case our constructions comple- 
ment the results on the tensor products of holomorphic and anti-holomorphic 
discrete series representations in jRj. 

For Riemannian symmetric spaces, and also for several non-Riemannian 
ones, we have m(rr) < 1. Thus in these cases we deduce that tt\ ® iT2 is multi- 
plicity free. 
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If fl is the Siegel upper half plane then G — Sp(2n, R), G is the metaplectic 
group and G'/H' = 0(p+ ,p-)/[0(pf ,p7/)x . . .xO(p+ ,pj)], where p+ = £>+ 
P=52p7- ^ n this case our correspondence coincides with the _ff'-spherical 
part of the ^-correspondence. 

Finally, if is the exceptional tube domain, then G = G is the simply 
connected exceptional group E 7 (_ 25 y If we take s — 2 and p\ — (1,0), then 
among the possibilities for G'/H' are the various forms of the Cayley projective 
plane (X[ p. 118], i.e., for p 2 — (2,0), p 2 = (0,2) and p 2 = (1,1), we obtain 
respectively 

F4(- 52 )/Spin(9), F 4{ _ 20 )/Spin(9) and F^ 2 o)/Spin(l, 8). 

Note that these symmetric spaces are multiplicity-free (see jv| for the non- 
Riemannian space F4(_ 20 j / Spin(l, 8)) and by Theorem 0.4, so is II = -k\ ® ir 2 . 

If we take p\ = p 2 = (1,0), then II is a tensor square of a highest weight 
representation tti , and a description of the spectrum of this tensor square is a 



key step in the classification of unitarizable highest weight modules in |EHW|. 

Just as with the ^-correspondence, we expect that our results will have 
smooth and global analogues. We shall take up some of these questions in 
subsequent papers. 



1 Notation and Preliminaries 

1.1 Groups and subgroups 

Let G be one of the following groups: 

• Sp(2n,R) (case II), 

• U (n, n) (case 12), 

• 0*{An) (case 13), 
. 0(2, j) (case 14), 

• £ , 7(-25) (case 15), 

and K be the maximal compact subgroup of G. Then Q = G/K is a sym- 



metric domain of tube type [He, p. 474]. Taking G = U(n,ri) or 0{2,j) instead 
of SU(n,n) or 50(2, j) is not really necessary, but will make some arguments 
more straightforward. 

The restricted root system for each of the groups listed above is of type 0„, 
where n is the real rank of the group G. Let A = {/3i, /3 2 , ...,/?„} be the basis 
of the restricted root system, enumerated in such a way that the corresponding 
Dynkin diagram is 

01 02 03 0n-2 0r,-l n 

o o o • ■ • o o <S^= o 
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There exists a one-to-one correspondence between the set of maximal parabolic 
subgroups of G and the set of maximal subsets of A. We will be interested in 
two parabolic subgroups of G - the Siegel parabolic of G (this corresponds to 
the set A \ {/3 n }) and the maximal parabolic subgroup corresponding to the set 
A \ {Pi}. We denote the first by P and the second by P' . 

The Levi decomposition of P is P = L ■ N , where the subgroup N is abelian 
(e.g., for G = 0(2, j) we have n = 2,L = R* x 0(1, j - 1) and N = R 1 '^ 1 ). 

The Langlands decomposition of P' is P' = M'AN', where the radical N' 
is a two-step nilpotent group with a one-dimensional center ZN' , and we can 
identify it with the real Heisenberg group of dimension 2m + 1. The vector 
subgroup A is one-dimensional, i.e., A — R*. For example, G = £V(-25) gi yes 
M'A = SO(2, 10) x R* and N' is the Heisenberg group associated with a 32- 
dimensional real vector space. 

The group M' splits into a direct product of a compact factor and a non- 
compact group, which we denote by G_. The group G_ belongs to one of the 
families (I1)-(I4), and we can consider its Siegel parabolic P_, the nilradical N- 
of P_ , etc. In general, all subgroups of G_ will be written with a minus as a 
subscript. 

The information about some of the subgroups we defined above is summa- 
rized in the following table. 



G 


N 


M' 


G- 


m 


Sp(2n,R) 


Sym(n,R) 


Sp(2n ~ 2, R) 


Sp(2n - 2, R) 


n-1 


U(n,n) 


Herman, C) 


(7(1) xU(n-l,n-l) 


U(n-l,n-l) 


2(n - 1) 


0*(4n) 


Herm(n,U) 


Sp(l) x 0*(4n-4) 


0*(4n - 4) 


4(n - 1) 


0(2,j) 




SL(2,R) x 0(J-2) 


SL(2,R) 


3-2 


-EV(-25) 


Herm(3,0) 


SO(2, 10) 


50(2,10) 


16 



1.2 Orbits and representations 

The orbits of the natural action of L on n* = N* are parametrized by pairs of 
non-negative integers p + ,p~ with p + + p~ < n |s|, 2.1]. The simplest example 
of this parametrization can be observed for G = Sp(2n, R), when the group N 
can be identified with the vector space of n x n real symmetric matrices, and 
an arbitrary orbit O of L = GL(n, R) on N* ~ 7Y is defined by the signature of 
the symmetric matrix £ £ O. We write p for the pair (p + ,p~), and O p for the 
corresponding orbit. The rank of the orbit is p + + p~ , which we denote by \p\. 
If \p\ = n, the orbit O p is open in ^V*, otherwise we get small (singular) orbits. 
By S p we denote the stabilizer of £ p 6 O p in L. 

Suppose now \p\ < n. Then O p = L/S p has an L-equivariant measure dfip 
which transforms by some positive character S p of L. The main result of p| 
associates with each nonzero singular orbit O p a unitary irreducible representa- 
tion tt p of G. Here G — G unless G = Sp(2n,M) or 0(2, j) with j odd and G 
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is a two-fold cover of G in these two cases (for G = Sp(2n, K) we can take G to 
be a real metaplectic group). 

If H is a subgroup of G, we write H for the inverse image of H in G. 

The representation ir p acts on the Hilbert space L 2 (O p , dp,p), and actions of 
the elements of the maximal parabolic subgroup P can be written in a particu- 
larly simple manner - the action of the reductive part L comes from the action 
of L on O p and the unipotent radical N acts by characters 

K{n)h]{^ p ) = x ? » h(Q, n e N,£ p e O p (3) 

Here \£ P is the unitary character of the vector space N defined by £ p £ N* and 
v p is a unitary character of L (trivial on the identity component of L)Q 



2 Tensor products ir Pl <8> . . . <8) 7r Ps 

We pick s singular orbits C Pl , . . . , C Ps such that [pi | + . . . + \p s \ < n and consider 
the tensor product of associated representations 

s 

n = (g)7r Pi . 

i=l 

The group L acts on the set O' = f C Pl x . . . x Ps , and up to a set of measure 
zero, O' is a single L-orbit. Note that the set 

O = O pl + . . . + Ps = {( E N* \ ( = E- =1 C Pi , C Pi G O w } 

also contains a dense L-orbit. The representation II acts in ®* =1 L 2 (0 Pi , dfi Pi ), 
and we can identify this space with L 2 (0' , dp!) where du! is the product measure. 
If we fix a generic representative £' = (£ Pl , . . . , £ Ps ) G O' and set 

f = &i + ••• + &. e O 

and 8 — Y[t=i 8 Pi , v = Yli=i v Pi , we have the following formulas for the actions 
of 1% on L 2 (0'\dp') 

nfl )/(io = KfoWor 172 ^ 1 ^'), ^ g i (4) 

n(no)/(JO = XifW^O, n €N . 

Let now 5" and S be the isotropy subgroups of £' and £, respectively, with 
respect to the action of L on C and O. If + . . . + \p s \ = n, the groups S' 
and S are reductive. 

1 In p| the characters u p and 5 P are denoted by /x and i/ respectively. 
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Example. Take G — U(n, n), s = 2 and p\ — (k, 0), P2 = (0, n — k). Then we 

can choose f Pl = ( ^ o )' ^ = ( -jJL ) and ^ = ( -/L )" 
It is easy to see that S = U(k, n — k) and S' = U(k) x U(n — k). The quotient 
5/ S' is a Riemannian symmetric space. 



Lemma 2.1 II |p ~ v ® Indg/jy^l <8> Xf) (£ 2 -induction) . _ffere v is a character 
of L extended trivially to P. 

Proof. We denote the induced representation v (E> Indg, N (l ® x$) by II'. Then 
by the definition of the induced representation, II' acts on the space C of square- 
summable functions satisfying a standard invariance condition 

C = {/ : LN -> C | / (ps'n) = xd n )f(p) for pe?,s'eS' n£ AT}. (5) 

Let |px I + • • • + \Ps | < n (strict inequality) . Then the quasi-invariant measure 
on the quotient space LN/S'N ~ L/S' is transformed by the character 5 = 

n'(Zo)/(Zn) = y{h)8{h)- l / 2 f{l^ln) (6) 
n'(no)/an) = /(no x /n) = /(((r'^ln)) . 

We can now define a unitary operator : L 2 (0' , du.') — > C by setting 

[*/i](fn)=xc(n)- 1 A(0- 

This operator provides an isometry between L 2 ((D' ,dfi') and C, and it easy to 
check that $ intertwines the actions II and II'. Indeed, for the actions of 1$ this 
is immediate by inspection of formulas (Jij) and (6), and for the actions of no we 
get 

W(n )[m](ln) = [*/i](/(r 1 n o - 1 Zn) = X |(r 1 n o - 1 0- 1 X S (n)- 1 /i(^ / ) 
= Xf(n) _1 *c(«o)ft(lO = [*n(no)ft](Jn). 

Computations for + . . . + \p s \ = n are almost identical. In this case the 
space L/S' possesses an L-invariant measure, the action of Iq £ L is given by 

n'(Z )/(Zn) - /(^n) , 

and we set 

m(in)=xd^r 1 m- 1/2 (^)- 

A straightforward computation shows that this operator intertwines the actions 
of n and IT. ■ 
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Denote by 7 a ^'-quasi-regular representation of S in L 2 (S/S'). That is 

( 7 (z)/)(x) = f(z- l x) for z £ S, x (E Y d ^ S/S', f e L 2 (Y). (7) 

Of course, 7 = Indf.,1, and combining the induction in stages with the fact that 
the character x$, of N is SW-fixed, we get 

1% ~ v®iod%,,r(l®xt) = ^«>Indfjv((Ind| 1)®%) = ^®Indf Ar ( 7 cg)x 4 ). (8) 

The groups S and 5" are, generally speaking, not reductive (except when \pi | + 
. . . + \p s \ = n). As was discussed in [S, 2.1], the Lie algebras s and s' of S and 
<S', respectively, can be written as 

s' = ([l + f)') + U, 

where [i,fl',r/ are some reductive Lie algebras, f)' C fl' and u is a nilpotent 
radical common for both s and s' . Let G' and H' be the corresponding Lie 
groups. 

In particular, X = G / H' is a reductive homogeneous space, and we can 
consider an H '-quasi-regular representation of G' on L 2 (X) (denoted by 7'). 
Then the representation 7 of S given by the formula (0) can be obtained by 
extending 7' trivially from G" to S. Now let 

7' ~ / m{ir)iT du,(ir) 



be a decomposition of a quasi-regular representation 7' into a direct integral of 
unitary irreducible representations of G", where m : G' — » Z + is a multiplicity 
function and (i/i a Planchcrel measure for a symmetric space X. Each irreducible 
representation n <E G' can be extended to an irreducible representation 7r v of S. 
This gives 

/•© 

m(7r)7r v d[i(Tr) 

G' 



and substituting this into (8), we obtain the decomposition of Theorem 3.3 



n|p~ / m(7r)e(ir)d//(7r), (9) 

where <d{ir) = v ® Ind|' A r(7r v ® x^). Note that representations 7r present in the 
formula (9) (i.e., those with m(7r) > 0) are ff '-spherical representations of G'. 

Mackey theory guarantees that all representations Q(tt) are unitary irre- 
ducible representations of P and Q(ir) ~ 0(c) if and only if tt ~ ct. 

The special case s = 2 deserves some special attention. In this situation 
II = tt p (g) 7r g , where p = {p + ,p~),q = (q + ,q~). We will write G' , if' and 
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X pq for G' , H' and X, respectively. The quotient space X pq = G' pq /H pq is then 
a reductive symmetric space in the sense of IF]. The table below lists these 
symmetric spaces for different combinations of G, p and q ( \p\ + \q\ < n) (see 
jA|, 16.7] for the detailed computations in the case of G = E-j). 



G 


P 


q 


Xpq 


Sp(2n, R) 


P 


q 


0(p + + q + ,p~ + q-)/[0(p + ,p-) X 0(g+ ,q~)] 


U(n, n) 


p 


q 


U(p + + q + ,p-+q-)/[U(p + ,p-) X U(q + ,q-)] 


0*(4n) 


p 


q 


Sp{p + + q + ,p~ + q-)/[Sp(p + ,p-) X Sp{q + ,q-)] 


0(2,j) 


(1,0) 


(1,0) 


sou - 1)/ sou - 2) 




(1,0) 


(0,1) 


SO (l,j-2)/SOU-2) 




(1,0) 


(1,0) 


SO(9)/SO(8) 




(1,0) 


(0,1) 


5O (l,8)/5O(8) 




(1,0) 


(2,0) 


F 4 (_52)/Spin(9) 




(1,0) 


(0,2) 


F 4 (-2o) / Spin(9) 




(1,0) 


(i,i) 


F 4 (- 20 ) /Spin(l,8) 



3 Extending 0(tt) to G 
3.1 The A^-spectrum 

In this section we study low-rank representations of G. For the classical groups 
(cases II, 12, 13 in our list) a complete theory of low-rank representations can 



be found in Li's paper [L2|. We rely heavily on the ideas and methods of this 
paper. Our objective here is to extend the low-rank theory of Li so it can be 
applied to representations of the groups 0(2,j) and £?7(-25)- 

Consider the restriction of the representation 6(71") = ^®Ind5 Ar (7r v (8>Xs) to 
N. This restriction decomposes into a direct integral of unitary characters, and 
the decomposition is determined by a projection- valued measure oniV = N*. 
This measure is supported on the set O C N* (an L-orbit of £). 

Similarly, for any unitary representation r of G, we can consider its restric- 
tion to the abelian subgroup N and the associated measure fi T on N*. If fi T is 
supported on the single orbit O r C N* , we say that r is of signature r = (r + , r~ ) 
and write 

sigrijv r = r. 

The number \r\ = r + + r~ is the rank of O r . If fi T is supported on one or 
several orbits of rank k, we write ranker t = k. 

(1,0),* >0 



It will be convenient to set signt = -> . 

\ (0,1),* <0 

Remark. For the representations of classical groups, the notion of rank was 



introduced in |Hol | and [L2|. Our definition extends it to G = E 7 /_2 5 y For 



G = 0(2, j) the definition above differs from the notion of rank in [L2] due to 
the different choice of the parabolic subgroup P. 
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We now take a unitary representation a of G and consider c\jp- N , 
group N' is a Heisenberg group defined by an exact sequence 



The 



1 -► ZN' -> N' 



1. 



and the multiplication on TV' defines a standard skew-symmetric bilinear form 
on R 2m . The group M' acts on N' by the automorphisms of N', and it also 
acts trivially on the center ZN'. Because of this we can view M' as a subgroup 
of Sp(2m,R). 

Now let pt be a unique representation of N' corresponding in the sense of 
Stone - von Neumann theorem to the character xt of ZN' ~ R, where 

Xt(z) = exp(2nitz), z E ZN 1 . 

We can extend pt to the representation of the semidirect product Sp(2m, R) -N' 
using the corresponding oscillator representation uj t of the metaplectic group 
Sp(2m, R) ~ . This extension restricts to a representation of a semidirect product 
M'N', and we denot e thi s restriction by pt- 

By the results of [ HM ] the subspace of Z TV-fixed vectors is invariant under 
the action of cr(G), and without loss of generality we may assume that a has 
no ZiV'-fixed vectors. Then, according to the Mackey theory, v\jp- N i decom- 
poses into representations of the form Kt ®p~t, 
representations of M' . We can write 



where all K t , t £ R* are unitary 



IG_ N 1 



K t ® pt dt. 



We will now describe the A^-spectrum of pt- It is known that a real vector space 
N is endowed with a structure of a simple formally real Jordan algebra with 
the unit (denoted by e), and L is the structure group of this Jordan algebra. 
Then ZN' is a one-dimensional subalgebra of N generated by the primitive 
idempotent c. 



This idempotent determines the Peirce decomposition of N [FK, IV. I] 
N = N(c, 1) + N(c, 1/2) + N(c, 0). 
Observe that N(c, 1) = ZN', N{c, 1/2) = R m and N(c, 0) = N-. 

Example. Take G = E n _ 25) . Then N = Herm(3, O), c 

the corresponding Peirce decomposition is 



1 





and 



N 



Ic-H 



+ Herm(2, 



»1,9 



Hence N(c, 1/2) = O 2 = R 16 and N(c, 0) = Herm(2, O) 
algebra is in fact the nilradical N- of the parabolic subgroup P. 
50(1, 9)] • R 1 - 9 inside G_ = 50(2, 10). 



and this Jordan 

fR* x 
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The action of pt on N(c, 1) and N(c, 1/2) is easy to describe: these spaces 
lie inside N' and we can model an irreducible representation p t of N' on the 
Hilbert space L 2 (N(c, 1/2)). To distinguish between an element x of N(c, 1/2) 
and the corresponding vector from WL m , we shall write x for the latter. Then 

Pt(nic)f(x) = Xt(ni)f(x) = Xt(tr N [n 1 c])f(x), m G K (10) 
Pt(n 1 / 2 )f(x) = Xt(ni/2 ■ x)f(x) = Xt(^tr N [n 1/2 x})f(x), n 1/2 G iV(c, 1/2). 

Here ir^v is the standard trace functional on the Jordan algebra N. 

Now take zo G Sym(m,M.) C Sp(2m,M). The action of the oscillator repre- 
sentation ivt(zo) on L 2 (R m ) is given by the formula 

ut(zo)f(x) = Xt(kxz x t )f{x). 

Observe that xzqx* = tr S y m ( m p)[(e - c)x 2 z }. 
Recall that G_ C Sp{2m,R) and 

N(c,0) = N_ = P_ n Sym(m, M). 

For n G iV(c, 0) and x G iV(c, 1/2) we have 

^sj,m(m,R) [(e - c)x 2 n ] = tr N _ [(e - c)x 2 n ] 

and 

Pt{n Q )f{x) = Xt{\tr N [{e - c)a; 2 r7, ])/(a;). (11) 

Combining formulas (10) and (|ll|), we can write the formula for p t (n°), 
where n° = n\c + 11,1/2 + 2no . 

Pt(n°)f(x) = X t(tr N ([c + \x + \{e - c)x 2 ]n°))f(x). (12) 

We can identify N* and N by setting <j>{n'){n") = tr N (n'n") for n,n' G N. It 
follows from the formula (12) that the iV-spectrum of pt is supported on the 
elements of the form nt(x), x G N(c, 1/2), where 

n t {x) = t(c + \x 4- \{e — c)x 2 ). 

For an arbitrary element x' G N(c, 1/2) there exists a special element of the 
structure group L, called the Frobenius transformation and denoted by t(x'). 
According to Lemma VI. 3.1 of [FK], r(x')n 4 (x) = n[ + n^ 2 + n' , where 

rij = tc 

n l/2 = t(2x'c+ |x) 

Hq = <(2(e - c)x /2 c + (e - c)x'x + \{e - c)x 2 ) . 
In particular, r(— §)rit(x) = tc. 
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We can now describe the ./V-spectrum of K t <8> Pt- If the 7V_-spcctrum of n t 
is supported on a set 0(K t ), then the support of the iV-spectrum of n t <S> pt 
consists of the elements n t {x) + n- where x G N(c, 1/2), n_ G 0{nt)- Then 

r(-|)(n t (x) + n_) = tc + n_ . 

Suppose now that sign N <j = r, where r = (r + ,r~). Then sign Ar (/« i <g> p t ) = r, 
i.e., 

sign w (te + n_) = r. (13) 
It is easy to sec that (13) implies signi + sign^ n_ = r, i.e., 

f (r+ - l,r"), t > 
Slgn ^- Kt = { (r+r-l),«0 ' 

We summarize this discussion in the following 

Lemma 3.1 Let a be a representation of G, sign^y a — r and <t\q_ n , — J Kt® 

R* 

p t dt. Then for any feK* the N --spectrum of the representation n t is supported 
on a single L --orbit, and sign^ K t = r — signt. 



3.2 Von Neumann algebras 

Let r be a representation of some subgroup H of G. By A(t, H ) we denote the 
von Neumann algebra generated by the operators r(h),h G H , where Hq is a 
subgroup of H. 

To proceed further we need 

Lemma 3.2 Assume that A(n t ,G-) = A(n t ,P-) for all tef*. Then 

© © 
A{ J K t ®p t dt,G-) ^Aj Kt®Ptdt,P-N r ). 



Proof. The representation p t is an irreducible representation of TV', therefore 
A(pt,N-) is the full algebra of bounded operators on L 2 (M. m ). Consider the 
algebra A{nt <8> Pt, G-). This algebra is generated by operators 

K t {g-)®p t {g-),g-eG-. (14) 

All these operators lie inside A(n t ® p t ,P-N'). Indeed, the algebra A(n t <8> 
p t ,P-N') contains the set B of all operators of the form K t (p~) <8> a, where a 
is an arbitrary bounded operator on L 2 (R m ) and p~ G P_. Combining this 
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fact with the assumption A(Kt,G~) = A(nt, P—)> we conclude that the von 



Neumann algebra generated by B already contains all operators (14). 
Hence A(n t (g> p { , G_) C A(n t ® p t ,P-N') and 

© © © 

A(J nt®Ptdt,G-) C J A{K t ®p tl G~)dt<z j ' A(Kt<g>p t ,P-N')dt . 

R* R* R* 

But the representations pt are irreducible and nonisomorphic for different t, and 

© 

A{Kt®pt,P-N')dt = A{( K t ®Ptdt,P-N'). ■ 
Observe that P_N' is a subgroup of P. 



The next theorem is an analogue of [L2, 4.3] 



Theorem 3.3 Let a be a representation of G , signer er = r and < \r\ < n 
(i.e., a is a low-rank representation of G). Then A(cr,G) = A(cr,P) ■ 

Proof. The groups G_ and P together generate G, and it suffices to check that 

A((T,G-)CA(a,P). (15) 

© 

But ct|q_ N i = J Kt <8> pt dt, and by Lemma 3.2, the assertion ( |l5| ) follows 
immediately if we can show that A(Kt,G-) — A(K t ,P-) for all t G K*. By 



Lemma B.l all K t are representations of rank |r| — 1 of the group G_, and we 
can apply the same line of reasoning to them. 

After \r\ steps of this process, we reduce the statement of the theorem to the 
case of representations of rank for some group Go, where Go is belongs to one 
of the families 11-14. Any representation r of rank decomposes over characters 



of Go [ HM ] and it is well known that any character of Go is determined by its 
restriction to the Siegel parabolic Pq (e.g., [jL2| , 4.2]). Therefore, A(t, Go) = 

a(t,p ). m 

We now return to the problem of decomposing representation LT = (££)f =1 Tr Pi ■ 
The restriction of this representation on P is given by (^|), and for any Q(tt) = 
v ® Ind5j V (7r v ®Xi) m the decomposition (0) 

sign^ 9(tt) = sign w i = Pi ■ 

Therefore II can be decomposed over the irreducible representations of G of 
signature J2UiP*- 
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Assume X)i=i \Pi\ < n - Then by Theorem 3.3, any two non-isomorphic 
irreducible representation from the spectrum of II restrict to non-isomorphic 
irreducible representations of P. Therefore the P-decomposition (^) gives rise 
to a G-decomposition 

n~ / m(n)0(w)dfi(n), (16) 
JG> 

where 9(tt) is defined for almost every ir (with respect to dfx) as a unique 
irreducible representation of G determined by the condition 8(ir)\-p = v ® 

Indf A r(7r v ® Xi)- Obviously 0(tt) ~ 9(a) if and only if n ~ a. 



4 Representations of maximal rank 

The statement of Theorem |3.3| is certainly false for the representations of max- 
imal possible rank, i.e., when sign N a — r and \r\ = n. Nevertheless, a G- 
decomposition ( |l^ ) can be constructed even when J2t=i \Pi\ = n - 

Consider a = ax ® a 2 , where a\ and a 2 are representations of G, sign^y a\ = 

© 

?'i = (r+,r~), |n | = n — 1 and signjy cr 2 = (1,0). Then (Ti\jf N , = J n t ®Ptdt 

© 

and a 2 \jj' N , — J k' u ® p u du, and 



(n t ® k' u ) ® (p t ® Pu) dt du. (17) 



For ( + w ^ we have (pt <8> Pu)\n' = Pt <8> Pu — 1 ® Pt+u, where 1 is a trivial 
representation of N' on L 2 (R m ). 

c~ ~ m / a i a f w+,i + it>0 . 

IPt ® Pu)\m' = u t ®uj u ~ w t)U ® w t „ where w tjU = <j , , ^,^ n and 

w U = 



w_,f + u < 

" '' ' Here ui+ and oj_ are the restrictions of two 



u-,tu/(t + u) < 

nonisomorphic oscillator representations of Sp(2m, R)~ to M . 

Then p t ® p u ~ r tjU ® pt+u, where T t , u (N') acts trivially on L 2 (M. m ) and 
Tt,«(M ) acts by „. The set i + u = has measure in R* x IR* and after a 
change of variables t + u = v the decomposition (17) becomes 



(n t ® n' v _ t <g> T t v -t) ® Pu dtdv, 



v 
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where V = {(t, v) \ t ^ 0, v ^ 0, v > t}. 

e 

If we set \ v = J Kt ® n'v-t ® T t,v-t dt, the preceding formula can be 



v\m'n> = f\v®Pvdv. (18) 



rewritten as 



By Lemma 3.1 all representations Kt have signature r*i — signi, and all n' v _ t 
are of rank 0, i.e., decomposable over characters. Therefore 

{k~ <g> u + , sign N _ k~ — (r+. r~ - 1) if v < 
(«- © (k+ ®w+), 

sign^ k~ = (r+,r _ - l),sign Ar _ k+ = (r+ - l,r - ) if t> > 0. 

(19) 

Remark. If the signature r is semi-definite (i.e. r — (|r| ,0) or (0, |r|)), some 
of the signatures in the formula above will involve negative numbers, which is of 
course impossible. To simplify notation, we agree that in this case corresponding 
summands are simply absent from the decomposition (18). 

Lemma 4.1 Let a = o\ ® 02, where o\ and 02 are representations of G = 
0(2, j), sigrijy (7i = r, \r\ = 1 and sign^ a% — (1,0). Then A(a,G) = A(a,P). 

Proof. Consider P 1 = P_ x 0(j — 2) - a parabolic subgroup of M' = SL(2) x 
0(j — 2). It suffices to prove that for all v 

A(X V ,M') = A(X v ,P t ). 



Indeed, this fact combined with formula (|lj) and Lemma 3^ gives A(a, M ) C 
A(<7, P), and the statement of the lemma follows. 

Analysis of (|l^) shows that X v \jj> = Xv ® where \v decomposes over 
characters and u> is an oscillator representation restricted to M'. Without loss 
of generality we may take u> = lo + . Two factors of M' form a dual reductive 
pair inside Sp(2(j — 2),R) and the spectrum of w+ is very well known: lu+ = (J^ 

T)i (8> i]^ , where 77^ and 772 are irreducible highest weight representations of 
SL(2) and 0(j — 2) respectively, and each r/i and ry^ occurs only once in the 
decomposition. Observe that each r)[ |p is irreducible. Therefore ^4(77^'' ® 

^ i} ,M') = A(t]^ (8^ i) ,P r ) ) and .4(^,1?') = y^P 1 "). Similarly, .4^ ® 
w, M ) = ^(xt,®^, P 1 ). Hence any irreducible component of Xv®u is irreducible 
when restricted to P 1 and uniquely determined by this restriction, and A(xv ® 

Remark. It is easy to see (by inspection of the above argument) that the 
statement of the lemma remains true if we replace a = o\ ® 02 with a = 
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®i=i ® a 2 l \ where sign^c^ = r and sign^cr^ = (1,0), 1 < i < k. We 
can also replace G = 0(2,j) with 50(2, j). 

Lemma 4.2 Let a — <Ji ® er 2 , where <j\ and <t 2 are representations of G = 
£ , 7(_25), signw °~\ = r : M — 2 and sign^y ct 2 = (1, 0). Then A(cr, G) = A(a, P). 

Proof. In this case M' = G- = SO(2, 10). Once again, it suffices to check that 
for all tiel* 

A(X V , G- ) = A(\ V ,P-). (20) 

From (19) we see that X v is either a tensor product of two representatio ns o f 
rank 1 (in this case the assertion of (|2^) follows immediately from Lemma 4.1 ) 
or 

A.„ = (k~ <8> w_) © (k+ ® 

where sign^ k~ = (1,0), sign^ k+ = (0, 1). This can occur only when r — 
(1, 1), v > 0. But signjy_ w_ = (0, 1), sign w _ uo + = (1, 0) and we find ourselves 



in a situation described in the remark to Lemma 4.1 
Therefore, (poh holds for all v. ■ 



Corollary 4.3 Let a = ir pi ® tt P2 <8> n p3 be a representation of G — £V(- 2 5), 
\pi\ = 1, \p 2 \ = 1,P3 = M ■ Then A(a,G)=A(a,P). 

Proof. Set (Ti = ir pi ®7r P2 , cr 2 = ir P3 . Then sign^y cti = pi +p 2 , sign^ ct 2 = (1,0) 
and the lemma above can be applied. H 

We now return to our study of the tensor product II = n Pi , Si=i Ip* I — 

n. 



Theorem 4.4 A{U,G) =A(IL,P). 

Proof. If X)i=i \Pi\ < n i the statement of this theorem follows from Theorem 
3.3. Hence we can restrict our attention to the case Xh=i \Pi\ = n - 

If G = 0(2, j), the only possible case is s = 2, and we may always assume 



p-2 = (1,0) and apply Lemma 4.1. Similarly, for G = -EV(_25) we can take 
p s = (1,0), and the theorem follows from Lemma [T^ for s — 2 and Corollary 
Ofor s = 3. 



Finally, in the classical cases (11-13) the statement follows immediately from 
[ |L2"| , 4.7-4.8]. Indeed, for these groups each of the representations ir Pi appears 
in the Howe duality correspondence for an appropriate stable range dual pair 
(GJ, G) and all irreducible representations from the spectrum of n appear in the 
duality correspondence for the pair (G',G), which is still in the stable range. 
Therefore any irreducible representation from the spectrum of n is irreducible 
when restricted to P and uniquely determined by this restriction, and -4(n, G) = 
A(U,P). U 
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Therefore the P-decomposition (^) gives rise to a G-decomposition of II with 
respect to the same measure dfj, and multiplicity function m(jr) 

n ~ / m(ir)e(w)dn(ir). (21) 

J G' 

Comparing (9) and ( |2l| ) we see that 9(n) is a unitary irreducible representa- 
tion of G which can be defined (for almost every tt with respect to d/i) as a 
unique irreducible representation from the spectrum of II satisfying the condi- 
tion 0(ir)\-p= Q(7r), where 9(ir) = v <g> Indf N (ir v <8>Xi)- 
Theorem 0.4 is thus proved. 
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